Issue 357: Nodejs ReaderGroup and EventReader #358

thekingofcity · 2022-03-11T14:00:14Z

Signed-off-by: thekingofcity 3353040+thekingofcity@users.noreply.github.com

Change log description
Add ReaderGroup and EventReader support to Nodejs bindings.

Purpose of the change
Fixes #357

What the code does
Implement the stream_reader_group and stream_reader just like the Python code.

It also uses the asynchronous approach so that users can do the followings:

const seg_slice: Slice = await stream_reader.get_segment_slice();
const enc = new TextDecoder('utf-8');
for (const event of seg_slice) {
    const raw_value: ArrayBuffer = event.data();
    console.log(`Event at ${event.offset()} reads ${enc.decode(raw_value)}`);
}

This should be merged after #355.

Reference about @@iterator:

How to verify it
Since the writer is not here, I use Python code to create the stream and write something to Pravega.

npm run build-debug
python3 tests/write.py will write 4 events.

b'e1'
b'e2'
b'e3'
b'e4'

node --loader ts-node/esm tests/stream_reader.ts should print these.

(node:5239) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
Event at 0 reads e1
Event at 10 reads e2
Event at 20 reads e3
Event at 30 reads e4

I will write similar tests like Python after the writer is implemented.

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

tkaitchuck · 2022-04-02T00:13:02Z

nodejs/src/stream_reader.rs

+    ///
+    /// Return the event offset in the segment.
+    ///
+    pub fn js_offset(mut cx: FunctionContext) -> JsResult<JsNumber> {


Do we want to expose this?

Not sure. Will exposing this cause confusion to users? The Python __repr__ implementation of EventData does expose the offset, so I'm assuming it's something that is ok to be exposed.

It's fine to pass it to the application and for them to pass it back to us via APIs. But we want to be clear that one cann't do pointer math with it. IE: It is not ok to generate your own integer based on this value and then use that to refer to something, because we don't guarantee that we won't have any header or compression or something in the future. So it's fine so long as the value is not interpreted.

In Java we do this by having an Object which wraps it which is opaque to users, so while it is just a long they can't directly get that long for visibility reasons. This is the normal pattern in Java to lock things down. In Python that is not the normal pattern and underscores are used in the name to indicate caution. I don't know what the norm is in JS. Is there a common pattern around this?

There is a keyword called readonly that works similarly to Python's underscore pattern, except you can not change the variable. But if you've written Python or JavaScript or any similar dynamic code, you would know users could access and manipulate anything they want. So instead of posing some restrictions on offset, it may be better to state it in doc or comment and let users be clear about what they are doing.

Besides, both py and js bindings don't allow users to specify a streamcut/offset on reading, so we may also improve this later.

nodejs/src/stream_reader.rs

nodejs/src/stream_reader_group.rs

nodejs/tests/stream_reader.ts

tkaitchuck

Clippy has some comments also

nodejs/tests/stream_reader.ts

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

thekingofcity · 2022-04-11T03:24:21Z

CI tests and clippy problems are fixed.

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

tkaitchuck

Approved with comments

nodejs/src/stream_manager.rs

tkaitchuck · 2022-04-21T00:20:52Z

nodejs/src/stream_reader.rs

+    ///
+    /// Return a Slice in an asynchronous call.
+    /// The actural returned type from await will be `Promise<Slice>` aka `JsResult<JsBoxed<RefCell<Slice>>>`.
+    ///
+    /// See the tokio-fetch example for more details on how to return a Promise and await.
+    /// https://github.com/neon-bindings/examples/tree/2dbbef55f483635d0118c20c9902bf4c6faa1ecc/examples/tokio-fetch
+    ///


Is this meant to be a user facing comment?

No, all comments in rust should not be a user facing comment/doc. Users are advised to read the README.md for a tutorial and https://pravega.github.io/pravega-client-rust/nodejs/index.html for a full API reference(auto generated from ts files and contains user facing comments.) The API reference is added in https://github.com/pravega/pravega-client-rust/pull/366/files#diff-d7ecc9db1d2f1f75440ba2c3e433aec8c252e9c86a6260732400737857c71d3b and should be available in the github.io pages after merge.

nodejs/src/stream_reader_group.rs

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

thekingofcity added 8 commits March 4, 2022 14:18

Issue 354: Nodejs StreamManager

cb28050

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

update doc to ///

5a2f8a8

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

favor composition over inheritance

678caf3

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

remove redundant hello

2f02576

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

lint

7407404

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

Issue357: Nodejs ReaderGroup and EventReader

eaf10c8

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

add header and arc

66ede75

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

fix doc

c92c04c

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

thekingofcity force-pushed the issue-357-Nodejs-ReaderGroup-and-EventReader branch from a12749f to c92c04c Compare March 15, 2022 08:13

tkaitchuck self-requested a review March 18, 2022 23:44

Merge branch 'master' into issue-357-Nodejs-ReaderGroup-and-EventReader

7031ef3

tkaitchuck reviewed Apr 2, 2022

View reviewed changes

tkaitchuck reviewed Apr 7, 2022

View reviewed changes

nodejs/tests/stream_reader.ts Show resolved Hide resolved

thekingofcity added 3 commits April 11, 2022 10:59

fix redundant clone

53275f1

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

fix clone_double_ref and absurd_extreme_comparison

d47edae

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

add ci test

b2114c8

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

thekingofcity added 3 commits April 11, 2022 11:43

fix CI test

05c354a

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

remove unnecessary todo

f77e5ce

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

update python in CI

a298bb4

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

tkaitchuck approved these changes Apr 21, 2022

View reviewed changes

thekingofcity and others added 3 commits April 25, 2022 10:27

display the internal error

5c9dc29

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

fix typo

64a5848

Signed-off-by: thekingofcity <3353040+thekingofcity@users.noreply.github.com>

Merge branch 'master' into issue-357-Nodejs-ReaderGroup-and-EventReader

af29b33

tkaitchuck approved these changes May 6, 2022

View reviewed changes

tkaitchuck merged commit fe6bcf2 into pravega:master May 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 357: Nodejs ReaderGroup and EventReader #358

Issue 357: Nodejs ReaderGroup and EventReader #358

thekingofcity commented Mar 11, 2022 •

edited

Loading

tkaitchuck Apr 2, 2022

thekingofcity Apr 2, 2022

tkaitchuck Apr 21, 2022

thekingofcity Apr 25, 2022 •

edited

Loading

tkaitchuck left a comment

thekingofcity commented Apr 11, 2022 •

edited

Loading

tkaitchuck left a comment

tkaitchuck Apr 21, 2022

thekingofcity Apr 25, 2022 •

edited

Loading

Issue 357: Nodejs ReaderGroup and EventReader #358

Issue 357: Nodejs ReaderGroup and EventReader #358

Conversation

thekingofcity commented Mar 11, 2022 • edited Loading

tkaitchuck Apr 2, 2022

Choose a reason for hiding this comment

thekingofcity Apr 2, 2022

Choose a reason for hiding this comment

tkaitchuck Apr 21, 2022

Choose a reason for hiding this comment

thekingofcity Apr 25, 2022 • edited Loading

Choose a reason for hiding this comment

tkaitchuck left a comment

Choose a reason for hiding this comment

thekingofcity commented Apr 11, 2022 • edited Loading

tkaitchuck left a comment

Choose a reason for hiding this comment

tkaitchuck Apr 21, 2022

Choose a reason for hiding this comment

thekingofcity Apr 25, 2022 • edited Loading

Choose a reason for hiding this comment

thekingofcity commented Mar 11, 2022 •

edited

Loading

thekingofcity Apr 25, 2022 •

edited

Loading

thekingofcity commented Apr 11, 2022 •

edited

Loading

thekingofcity Apr 25, 2022 •

edited

Loading